Phase transition of the largest eigenvalue for non-null 
complex sample covariance matrices 

Jinho Baik * Gerard Ben Arous^^and Sandrine Pech# 

February 1, 2008 

Abstract 

We compute the limiting distributions of the largest eigenvalue of a complex Gaussian sample 
covariance matrix when both the number of samples and the number of variables in each sample 
become large. When all but finitely many, say r, eigenvalues of the covariance matrix are 
the same, the dependence of the limiting distribution of the largest eigenvalue of the sample 
covariance matrix on those distinguished r eigenvalues of the covariance matrix is completely 
characterized in terms of an infinite sequence of new distribution functions that generalize 
the Tracy- Widom distributions of the random matrix theory. Especially a phase transition 
phenomena is observed. Our results also apply to a last passage percolation model and a 
queuing model. 

1 Introduction 

Consider M independent, identically distributed samples yi, . . . , yM, all of which are x 1 column 
vectors. We further assume that the sample vectors y^ are Gaussian with mean /2 and covariance 
S, where S is a fixed N x N positive matrix; the density of a sample y is 

(27r)^/2(detS)i/2" ' 

where <,> denotes the inner product of vectors. We denote by ii,...,i]\f the eigenvalues of 

the covariance matrix S, called the 'population eigenvalues'. The sample mean Y is defined by 

*Department of Mathematics, University of Michigan, Ann Arbor, MI, 48109, USA, baik@umich.edu 
^Department of Mathematics, Courant Institute of Mathematical Sciences, New York, NY, 10012, USA, be- 
narous@cims.nyu.edu 

■'■Department of Mathematics Ecole Polyechnique Federale de Lausanne, 1015 Lausanne S-witzerland, san- 
drine. peche@epfl.ch; Current address: Institut Fourier, UJF Grenoble 38000 France, sandrine.peche@ujf-grenoble.fr 
§MSC 2000 Subject Classification: 15A52, 41A60, 60F99, 62E20, 62H20 

^Keywords and phrases: sample covariance, limit theorem, Tracy- Widom distribution. Airy kernel, random matrix 



1 



Y := jj{yi + ■ ■ ■ + yM) and we set X = [yi — Y , ■ ■ ■ , yM — Y] to be the (centered) N x M sample 
matrix. Let 5 = jjXX' be the sample covariance matrix. The eigenvalues of S, called the 'sample 
eigenvalues', are denoted by Ai > A2 > • • • > Ajv > 0. (The eigenvalues are simple with probability 
1.) The probability space of Aj's is sometimes called the Wishart ensemble (see e.g. j29jl. 

Contrary to the traditional assumptions, it is of current interest to study the case when is of 
same order as M. Indeed when T, = I (null-case), several results are known. As A^, M — > 00 such 
that M/7V ^ 7^ > 1, the following holds. 

• Density of eigenvalues [27] : For any real x, 

^#{A, : A, <x}^ H{x) (2) 

where 

^2 

H'ix) = —Jih - x){x - a), a<x<h, (3) 

2'KX 

and a = (3^)2 ^ndb={^f. 

• Limit of the largest eigenvalue (15j : 

A,^(l±^)' a.s. (4) 

• Limiting distribution [22l: For any real x, 

<(A.-{^rt-^^^<^)-W.) (5) 

where Fqoe{x) is the so-called GOE Tracy- Widom distribution, which is the limiting dis- 
tribution of the largest eigenvalue of a random real symmetric matrix from the Gaussian 
orthogonal ensemble (GOE) as the size of the matrix tends to infinity j4Uj . 

• Robustness to models |35j : It turned out that the Gaussian assumption is unnecessary 
and a result similar to © still holds for a quite general class of independent, identically 
distributed random samples. 

From and we find that the largest sample eigenvalue Ai in the null case converges to 

the rightmost edge of support of the limiting density of eigenvalues. However, in practice (see e.g. 
j23j ) there often are statistical data for which one or several large sample eigenvalues are separated 
from the bulk of the eigenvalues. For instance, see Figure 1 and 2 of the paper which plot 
the sample eigenvalues of the functional data consisting of a speech dataset of 162 instances of a 
phoneme "del" spoken by males calculated at 256 points [U]. Other examples of similar phenomena 
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include mathematical finance [HI] , HH] , US] j wireless communication |37j , physics of mixture , 
and data analysis and statistical learning JH]- As suggested in [231, ^^i^ situation poses a natural 
question: when S 7^ / (non-null case), how do a few large sample eigenvalues depend on the 
population eigenvalues? More concretely, if there are a few large population eigenvalues, do they 
pull to the sample eigenvalues, and for it to happen, how large the population eigenvalues should 



Though this might be a challenging problem for real sample data, it turned out that one could 
answer some of the above questions in great detail for complex Gaussian samples. Complex sample 
covariance matrix has an application in multi-antenna Gaussian channels in wireless communication 
P7j . Also the results of complex case lead us to a guess for aspects of the real case (see Conjecture in 
section [T31 below) . Another reason of studying complex sample covariance matrix is its relation to 
a last passage percolation model and a queueing theory. See section |H1 below for such a connection. 

Before we present our work, we first summarize some known results for the complex sample 
covariance matrices. 

1.1 Some known results for the eigenvalues of complex sample covariance ma- 



We assume that the samples y are complex Gaussian with mean fl and covariance S. Hence the 
density of y is precisely given by with the understanding that < , > denotes now the complex 
inner product. The (centered) sample matrix X and the sample covariance matrix S = jjXX* are 
defined as before where X* is the transpose followed by the complex conjugation. Recall that the 
eigenvalues of S, sample eigenvalues, are denoted by Ai > • • • > Atv > 0, and the eigenvalues of S, 
population eigenvalues, are denoted by £1, . . . ,£Ar > 0. 

• Density of eigenvalues [^Tj, [HI (see also Theorem 3.4 of |2|): When all but finitely many 
eigenvalues Ij of S are equal to 1, as M,N 00 such that M/N > 1, the limiting 

density of the sample eigenvalues Xj is given by 



be? 



trices 



^#{A, : A, < x} ^ H{x) 



(6) 



where H{x) is again defined by ©• 



• Null case : When S = /, as M, iV ^ 00 such that M /N j 



2 > 1, [ISI 




(7) 



and for any real x (see, e.g. ^3], jH]) 
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where Fgue{x) is the GUE Tracy- Widom distribution, which is the hmiting distribution of 
the largest eigenvalue of a random complex Hermitian matrix from the Gaussian unitary 
ensemble (GUE) as the size of the matrix tends to infinity Moreover, the limit © holds 
true for a quite general class of independent, identically distributed random samples, after 
suitable scahng |SS]- 

Remark 1.1. The distribution function Fgue is different from Fgoe- A formula of Fqije{x) is 
given in ()18|) below and a formula for {FcoEi"^))^ is given in ()24() below. 

Remark 1.2. When S = /, the probability space of the eigenvalues \j of S is sometimes called 
the Laguerre unitary ensemble (LUE) since the correlation functions of \j can be represented in 
terms of Laguerre polynomials. Similarly, for real samples with S = /, the probability space of the 
eigenvalues of S is called the Laguerre orthogonal ensemble (LOE). See e.g. P^ . 

Note that the limiting density of the eigenvalues Aj is known for general S 7^ /, but the 
convergence ©/(jSJ of Ai to the edge of the support of the limiting distribution of the eigenvalues 
was obtained only when S = /. The following result of Peche [S3 generalizes (jH)) and shows that 
when all but finitely many eigenvalues 1^ of S are 1 and those distinguished eigenvalues are "not 
too big", Ai is still not separated from the rest of the eigenvalues. 

• When iy+i = • • • = = 1 for a fixed integer r and ii = ■ ■ ■ = Ir < 2 are fixed, as 



A natural question is then whether the upper bound 2 of £1 = •••=£,. is critical. One of our 
result in this paper is that it is indeed the critical value. Moreover, we find that if some of ij are 
precisely equal to the critical value, then the limiting distribution is changed to something new. 



the Gaussian type order vM. In order to state our results, we first need some definitions. 



1.2 Definitions of some distribution functions 
1.2.1 Airy-type distributions 

Let Ai{u) be the Airy function which has the integral representation 



M = iV ^ cx), ISU 



P((Ai - 4) • 2-4/3M2/3 <x)^ Fgue{x). 



(9) 



And if one or more ^j are bigger than the critical value, the fluctuation order M^/^ is changed to 





(10) 



where the contour is from ooe^*'^/^ to ooe' 



,«7r/6_ jjgfing Airy kernel (see e.g. [35]) by 

Ai{u)Ai'{v) - Ai'{u)Ai{v) 



A{u,v) 



(11) 



u — V 
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and let A^,. be the operator acting on L^((x,cxo)) with kernel A{u,v). An alternative formula of 
the Airy kernel is 

A{u,v)= / Ai{u + z)Ai{z + v)dz, (12) 
Jo 

which can be checked directly by using the relation Ai"{u) = uAi{u) and integrating by parts. For 
m = 1, 2, 3, . . . , set 

= — [ e^^'^+^^'^'-Vda (13) 
^ ^ 27r y (m)"* ^ ^ 

where the contour is from c«e^*'^/^ to ooe*'^/^ such that the point a = lies above the contour. Also 
set 

tM(^;) = — f e'^'^+'^^'i-iar-^da (14) 
2vr J 

where the contour is from ooe^*'^/^ to ooe*'^/^. Alternatively, 

^+3n=m-l V / ^oo >) 

£,n=0,l,2,... 



and 

iM(^;) = ( --f ) Ai(i;). (16) 



(17) 



See Lemma 13.31 below for the proof that the two formulas of s^"^\u) are the same. 
Definition 1.1. For k = 1,2, . . . , define for real x, 

Fkix) = det(l - A J • det( 6mn- < —i— >) 

\ i Ax / l<m,n<k 

where <, > denotes the (real) inner product of functions in L?'{{x, oo)). Let Fq{x) = det(l — A^;). 

The fact that the inner product in p7j) makes sense and hence Fk{x) is well-defined is proved 
in Lemma 13.31 below. 

It is well-known that (see e.g. ^1], [SHI) 

Fo{x) = det(l - A,) = Fgue{x) (18) 

and hence Fq is the GUE Tracy- Widom distribution function. There is an alternative expression 
of Fq. Let u{x) be the solution to the Painleve II equation 

u" = 2u^ + xu (19) 

satisfying the condition 

u{x) ~ -Ai{x), x +00. (20) 



There is a unique, global solution ^7j, and satisfies (see e.g. ^7j, pUj l 

nix) = + as. ^+00 (21) 

u{x) = -J^{l + 0{x^'^)) asx^-00. (22) 



Then ^ 



Foix) = det(l - AW) = exp(^- ^ (y - x)n2(y)dy^ , 



(23) 



In addition to being a beautiful identity, the right-hand-side of (|23)) provides a practical formula 
to plot the graph of Fq. 

For /c = 1, it is known that (see [H], (3.34) of |H1) 

Fi(x) = det(l - A,).(^l- < >^ = {FGOE{x)f. (24) 

The function Fqqe also has a Painleve formula |4Uj and 

Fi(x) =Fo(2;)exp(^^ ^^(y)dy^ • (25) 

The functions F/^, k > 2, seem to be new. The Painleve formula of F/^ for general k > 2 will 
be presented in For each k > 2, Fk{x) is clearly a continuous function in x. Being a limit 
of non-decreasing functions as Theorem 11.11 below shows, Fk{x) is a non-decreasing function. It 
is also not difficult to check by using a steepest-descent analysis that Fk{x) — > 1 as x — > -|-oo (cf. 
Proof of Lemma l3.3|) . However, the proof that F^^{x) — > as x — > —00 is not trivial. The fact that 
Fk{x) — > as X — > — cxD is obtained in ^ using the Painleve formula. Therefore Fk{x), k > 2, are 
distribution functions, which generalize the Tracy- Widom distribution functions. (The functions 
Fq,Fi are known to be distribution functions.) 

1.2.2 Finite GUE distributions 

Consider the density of k particles ; • • • ) Cfc on the real line defined by 



p(6,...,e.)=^ n i^-^.f -rie"^^^' (26) 

l<i<j<k 1=1 

where is the normalization constant, 

/oo poo ^ 

•••/ n \^^-^l?■T{^-'^^^di,■■■diu = {27^?''X{j\, (27) 



l<i<j<k i=l j=l 
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which is called the Selberg's integral (see e.g. |2H])- This is the density of the eigenvalues of the 
Gaussian unitary ensemble (GUE), the probability space of k x k Hermitian matrices H whose 
entries are independent Gaussian random variables with mean and standard deviation 1 for the 
diagonal entries, and mean and standard deviation 1/2 for each of the real and complex parts of 
the off-diagonal entries (see e.g. 



Definition 1.2. For k = 1,2,3, ... , define the distribution Gk{x) by 

Gk{x) = — •••/ n \^i-Cj\^-ll^~~'^^d^i---dCk. (28) 



l<i<i<A: 1=1 



In other words, Gk is the distribution of the largest eigenvalue oi k x k GUE. When k = 1, this 
is the Gaussian distribution, 

1 1 

Gi{x) = ^= / e-2^'dCi = erf{x). (29) 
V27r J-oo 

There is an alternative expression of Gk in terms of a Fredholm determinant similar to the formula 
((T7|) of Fk- Let Pnix) = CnX^ + • • • be the polynomial of degree n {cn > is the leading coefficient) 
determined by the orthogonality condition 



oo 



Pm{x)Pn{x)e 2^' dx = 5r,in. (30) 

The orthonormal polynomial pn is given by 

where Hn is the Hermite polynomial. The leading coefficient Cn of p„ is (see e.g. j24j ) 

Cn = 1 — (32) 

(27r)V4^ ^ ' 

Then the so-called orthogonal polynomial method in the random matrix theory establishes that: 
Lemma 1.1. For any k = 1,2, . . . and x G M, 

Gfe(2;) =det(l-Hi*^)), (33) 
where is the operator acting on L'^{{x,oo)) defined by the kernel 

H(*^)(7i v) = ^fc-l Pkiu)pk-l{v) - Pk-l{u)pk{v) ^_(„2^^2)/4 

This is a standard result in the theory of random matrices. The proof can be found, for example, 
in j2H], j41j . There is also an identity of the form (|23j) for det(l — H^^), now in terms of Painleve 
IV equation. See jSHl- 
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1.3 Main Results 

We are now ready to state our main results. 

Theorem 1.1. Let Ai be the largest eigenvalue of the sample covariance matrix constructed from M 
independent, identically distributed complex Gaussian sample vectors of N variables. Let ii, ■ ■ ■ ,iisf 
denote the eigenvalues of the covariance matrix of the samples. Suppose that for a fixed integer 
r > 0, 

ir+l = er+2 = ---=iN = l. (35) 

As M, N ^ oo while M/N = 7^ is in a compact subset of [1, 00), the following holds for any real 
x in a compact set. 

(a) When for some < k < r, 

£i = ---=4 = l+7"' (36) 
and 4+1, . . . ,ir are in a compact subset of {0,1 + 

p((Ai - (1 + 7-^)^) • jYTW^^'^' - ^ ^^^^ 

where F}^{x) is defined in (jl7jl . 

(b) When for some 1 < k < r, 

ii = ■ ■ ■ = ik is in a compact set of (1 + 7"^, 00) (38) 
and £k+i, ■ ■ ■ ,ir o.re in a compact subset of (0, £1), 

- + 1^)) ■ . , a .) ^ G,i.) (39) 

\Jh (£1-1)^ 

where Gk{x) is defined in (|28|1. 
Hence, for instance, when r = 2, 

h = --- = iN = l (40) 

and there are two distinguished eigenvalues ii and £2 of the covariance matrix. Assume without 
loss of generality that ^l > £2- Then 



Fo{x), < £1,4 < 1+7-1 
Fi{x), 0<£2 < 1 + 7"^ =^1 (41) 
^F2ix), 4 =£2 = 1 + 7"^ 



and 



M 



< X 



G2{x), i^ = e2>i + -f-\ 



Jf2_Jhfl 

V 1 (^1-1)- 

assuming that £i,£2 are in compact sets in each case. See Figure ^ for a diagram. 



(42) 



71, A 



A 


G, . 






Fi 


Fi 

Gi 


Fo 


-Fi 

> 







Figure 1: Diagram of the Umiting distributions for various choices of li = tt-^^ and ^2 = "^2 ^ while 

£3 = ---=iN = l- 

Note the different fluctuation orders M'^^^ and y/M depending on the values of ^1, ^2- This type 
of 'phase transition' was also observed in [3 El EB] for different models in combinatorics and last 
passage percolation, in which a few limiting distribution functions were also computed depending 
on parameters. But the functions F^, /c > 2, in Theorem 11.11 seem to be new in this paper. The 
last passage percolation model considered in [Sj 03] has some relevance to our problem; see Section 
ini below. 

Theorem 11.11 and the fact that and Gk are distribution functions yield the following conse- 
quence. 

Corollary 1.1. Under the same assumption of Theorem M.lV the following holds, 
(a) When for some < k < r 



£i = --- = 4 = l+7^S 
and £k+i, ■ ■ ■ , 4 are in a compact subset 0/ (0, 1 + 7""*^), 

Ai — > (1 + 7~^)^ in probability. 



(43) 



(44) 
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(b) When for some 1 < k < r, 

^1 = • • • = 4 > 1 + 7"' (45) 
and ik+i, ■ ■ ■ ,ir o.re in a compact subset of (0, £i), 

/ 7-2 \ 

Ai — > 4 ( 1 + ^ 1 i'Ti' probability. (46) 

Proof. Suppose we are in the case of (a). For any fixed e > and x G M, 

limsupP(Ai < (l-e)(l+ 7-1)2) < li^nsupP Ai < (1 + 7'')'+ , , = Fk{x). (47) 
By taking x — > —00, we find that 

Urn P(Ai < (1 - e)(l + 7-^)^) = 0. (48) 

Similar arguments implies that P(Ai > (1 + e)(l + 7-^)^) 0. The case of (b) follows from the 
same argument. □ 

Together with Theorem ll . 1^ /Corollarv im imply that under the Gaussian assumption, when 
all but finitely many eigenvalues of S are 1, Ai is separated from the rest of eigenvalues if and only 
if at least one eigenvalue of S is greater than 1 + 7""*^. Theorem 11.11 also claims that when Ai is 
separated from the rest, the fluctuation of Ai is of order M^/^ rather than M^/^. Here the critical 
value 1 + 7-1 comes from a detail of computations and we do not have an intuitive reason yet. 
However, see Section El below for a heuristic argument from a last passage percolation model. 

Compare the case (b) of Theorem ll.l| /Corollarv 11.11 with the following result for samples of 
finite number of variables. 

Proposition 1.1. Suppose that there are M samples of N = k variables. Assume that all the 
eigenvalues of the covariance matrix are the same; 

h = ---=ik- (49) 

Then for fixed N = k, as M ^ 00, 

lim P( (Ai -4)^VMx ) = Gfc(x) (50) 

M—*oo \ li J 

and 

Ai — > li in probability. (51) 
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This result shows that the model in the case (b) of Theorem ll.l| /Corollarv 11.11 is not entirely 
dominated by the distinguished eigenvalues ii = ■ ■ ■ = ik the covariance matrix. Instead the 
contribution to Ai comes from both ii = ■ ■ ■ = ik and infinitely many unit eigenvalues. The proof 
of Proposition II. II is given in section |SJ 

Further detailed analysis along the line of this paper would yield the convergence of the moments 
of Ai under the scaling of Theorem ll.il This will be presented somewhere else. 

The real question is the real sample covariance. In the null cases, by comparing (jSJ and we 
note that even though the limiting distributions are different, the scalings are identical. In view of 
this, we conjecture the following: 

Conjecture. For real sample covariance, the Theorem 11.11 still holds true for different limiting 
distributions but with the same scaling. In particular, the critical value of distinguished eigenvalues 
£j of the covariance matrix is again expected to be 1 + 7"^. 

1.4 Around the transition point; interpolating distributions 

We also investigate the nature of the transition at £j = 1 + 7"^. The following result shows that if 
ij themselves scale properly in M, there are interpolating limiting distributions. 
We first need more definitions. For m = 1, 2, 3, ... , and for wi, . . . , Wm G C, set 

If m 

s(^'>{n;w^,...,wm) = — e'--+^-yYl —da (52) 

•> j=i 3 ^ 

where the contour is from ooe^^'^^^ to ooe*'^/^ such that the points a = iwi, . . . ,iwm lie above the 
contour. Also set 

„ m— 1 

t('")(t;; . . . , w^^i) = — e^'>+'¥' J] [w, - ih) db (53) 
where the contour is from ooe^*'^/^ to ooe*'^/^. 

Definition 1.3. For k = 1,2,..., define for real x and wi, . . . , Wk, 

Fk{x;wi,... ,Wk) 

(54) 



= det(l- A^) -detj^l- < ^ gM {wi, . . . ,Wn,), t^") {wi,..., Wn-i) >^ 
The function Fi{x;w) previously appeared in ^3] in a disguised form. See (4.18) and (4.12) of 



l<m,n<k 



m 
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The formula H54|) may seem to depend on the ordering of the parameters wi, . . . ,Wk- But as 
the following result (|5(ijl shows, it is independent of the ordering of the parameters. This can also 
be seen from a formula of |^. 

Like -Ffc, it is not difficult to check that the function F/^^x^wi, . . . ,Wk) is continuous, non- 
decreasing and becomes 1 as x — > -|-cxd. The proof that Fk{x;wi, . . . ,Wk) ^ as x ^ —oo is in 
Therefore, Fk{x;wi, . . . , Wk) is a distribution function. It is direct to check that Fk{x; wi, . . . , Wk) in- 
terpolates Fo{x), ... , Fk{x). For example, lim^j^+oo ^2(2;, 0, W2) = Fi{x), lim^j^+00 lim^2-++oo ^2(2;; wi,W2) 
Fq{x), etc. 

Theorem 1.2. Suppose that for a fixed r, ir+i = ^r+2 = • • • = -^at = 1- Set for some I < k < r, 

^.■ = ' + 7-- '^;;X> . ; = l,2,...,t. (55) 

When Wj, 1 < j < k, is in a compact subset o/M, and ij, k + 1 < j < r, is in a compact subset of 
(0, 1 -|- 7^^), as M, N ^ 00 such that M/N = 7^ is in a compact subset of [1, 00), 

Ip((Ai-(l+7"')')-^Yq:^^M2/3<x) -^Fk{x;wi,...,Wk) (56) 

for any x in a compact subset o/M. 

The Painleve Il-type expression for Fk{x] wi, . . . , Wk) will be presented in |^. 

This paper is organized as follows. The basic algebraic formula of the distribution of Ai in 
terms of a Fredholm determinant is given in Sectional where an outline of the asymptotic analysis 
of the Fredholm determinant is also presented. The proofs of Theorem 11.11 (a) and Theorem 11.21 
are given in Section El The proof of Theorem 11.11 (b) is in Section ^ and the proof of Proposition 
II. H is presented in Section |SJ In Sectional we indicate a connection between the sample covariance 
matrices, and a last passage percolation model and also a queueing theory. 

Notational Remark. Throughout the paper, we set 

7r,-=^7\ (57) 
This is only because the formulas below involving become simpler with vrj. 

Acknowledgments. Special thanks is due to Iain Johnstone for kindly explaining the importance 
of computing the largest eigenvalue distribution for non-null covariance case and also for his constant 
interest and encouragement. We would like to thank Kurt Johansson for sharing with us his proof 
of Proposition 12.11 below, Eric Rains for useful discussions and also Debashis Paul for finding a 
typographical error in the main theorem in an earlier draft. The work of the first author was 
supported in part by NSF Grant # DMS-0350729. 
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2 Basic formulas 



Notational Remark. The notation V{x) denotes the Vandermonde determinant 

V{x) = J{{xi - X,) (58) 

of a (finite) sequence x = (xi, X2, • • • )• 

2.1 Eigenvalue density; algebraic formula 

For complex Gaussian samples, the density of the sample covariance matrix is already known to 
Wishart around 1928 (see e.g. ^[): 

p(S) = ie-^^-*^(^"'^)(det5)*-^"^ (59) 

for some normaliztion constant C > 0. As S and S are Hermitian, we can set T, = UDU^^ and 
5 = HLH^^ where U and H are unitary matrices, D = diag{li, ■ ■ ■ ,In) = diag{'7T^^ , ■ ■ ■ ,vr^^) 
and L = diag{Xi, ■ ■ ■ ,Xn)- By taking the change of variables S i— > {L,H) using the Jacobian 
formula dS = cV{L)^dLdH for some constant c > 0, and then integrating over H, the density of 
the eigenvalues is (see, e.g. [T^ l 

1 ^ r 

p(A) = ±V{Xf TT Af • / e-^-*'-(^"Q^Q")dQ. (60) 

for some (new) constant C > where U (N) is the set oi N x N unitary matrices and A = 
(Ai, . . . , Aat). The last integral is known as Harish-Chandra-Itzykson-Zuber integral (see e.g. |28j ) 
and we find 

pw = n (61) 

Here when some of vTj's coincide, we interpret the formula using the I'Hopital's rule. We note that 
for a real sample covariance matrix, it is not known if the corresponding integral over the orthogonal 
group 0{N) is computable as above. Instead one usually define hypergeometric functions of matrix 
argument and study their algebraic properties (see e.g. EHl)- Consequently, the techniques below 
that we will use for the density of the form ()6H) is not applicable to real sample matrices. 

For the density H61() . the distribution function of the largest eigenvalue Ai can be expressed in 
terms of a Fredholm determinant, which will be the starting point of our asymptotic analysis. The 
following result can be obtained by suitably re-interpreting and taking a limit of a result of |3Uj . 
A different proof is given in [SJ. For the convenience of reader we include yet another proof by 
Johansson [^U] which uses an idea from random matrix theory (see e.g. j41|). 
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or 



Proposition 2.1. For any fixed q satisfying < g < min{7rj}jL;^, let 'i^M,N\{i^^oo) operat 
acting on L^((^,oo)) with kernel 

N 

KmMvX) = 7:^ [dzf rf„;e-^^(-^)+C^'^(-^)^(^)^^ n (62) 

where T, is a simple closed contour enclosing and lying in {w : Re{w) < q}, and T is a simple 
closed contour enclosing vri, . . . jTTn and lying {z : Re[z) > q}, both oriented counter-clockwise (see 
Figure\^. Then for any ^ G M, 

F(Ai < = det(l - KM,7v|(e,oo))- (63) 




Figure 2: Contours T and T, 



Remark 2.1. Note that the left-hand-side of H63() does not depend on the parameter q. The Fredhohn 
determinant on the right-hand-side of (|63|) is also independent of the choice of q as long as < 
q < min{7rj}^;^. If we use the notation Kg to denote K.m,n\{^,oo) for the parameter q, then 
Kg/ = EKgE"^ where E is the multiplication by e^"?'-"?)^; (E/)(A) = e('?'-'?)^/(A). But determinants 
are invariant under conjugations as long as both Kg' and EK^E"^ are in the trace class, which is 
the case when < q,q' < min{7rj}^^. The parameter q ensures that the kernel Kjv/,7v(??) C) is finite 
when r/ — > +00 or C — > +00 and the operator K.m,n\(^,oo) is trace class. It also helps the proof of 
the convergence of the operator in the next section. 

Proof. For a moment we assume that all vr^'s are distinct. Note that the density (|61|) is symmetric 
in Aj's. Hence using V{X) = det(Ai.~^), we find that 

1 poo poo ^ 

P(Ai < = 7^ / • • • / det(Ar') det(e-*^-^^'=) " X(c,oo)(Afc))Af "^dA^ (64) 
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with some constant C > 0, where X(g,oo) denotes the characteristic function (indicator function). 
Using the fundamental identity which dates back to 

j--- j det(/j(xfc))det(5(j(xfc)) J|c?/u(2;fc) = det^y fj{x)g,,{x)dfi{x)^, (65) 



we find 

P(Ai < = 7^ det f /"(I - X(5,oo)(A))A^-i+^^"^e"*^-^^dA') . (66) 

^ \J0 J l<j,k<N 

Now set u = M - N, (j)j{X) = xJ^^+'^e"^^"^ and = e-^^('^fe-9)A foj. ^ny q such that <q< 

min{7rj}^j^. Also let 

A = iAjk)i<j,k<N, A,u = <A,(A)$fc(A)dA = ^^l^^jt - (67) 
A direct computation shows that 

d«*^=na^5rT^-^^*^^^^^^ n ((Mvr,)-^ - (M.,)-^). (68) 

j=l ^ ^' j=l ^ ^' l<j<k<N 

Thus ^ is invertible. Also define the operators S : ^^((0,00)) ^^({1,. . ■,N}), C : . . .,N}) - 

L2((0,oo)) by 

i?(j,A)=0,(A), C{X,k) = <^k{X) (69) 
and let be the projection from (0,cx)) to (^,00). Then as 

X(5,oo)(A)A^-^+^^-^e-*^-^-^dA = {BP^C)ij,k), (70) 
we find that 

PiXi<0 = ^det{A-BP^C). (71) 

So, 

P(Ai < e) = det(l - A-^BP^C) = C" det(l - P^CA-^B) = C" det(l - P^CA-^BP^) (72) 

for some constant C" which does not depend on ^. But by letting ^ — > +00 in both sides of ()72() . 
we easily find that C" = 1. The kernel of the operator in the determinant is 

TV 

{CA'^B){r^, = E C{n,j)[A-^B){j, (), r/, C > (73) 



and from the Cramer's rule, 
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where A^^\(^) is the matrix given by A with jth column replaced by the vector {(f>i{(^), ■ ■ ■ , (/>jv(^))'^. 
To compute A^^\ note (the Hankel's formula for Gamma function) that for a positive integer a 

1 f 1 1 

_ / —dw = , = (75) 

2m J ^w" (a-1)! r(a) ^ ' 

where S is any simple closed contour enclosing the origin with counter-clockwise orientation. By 
replacing w — > C,Mw and setting a = j + this implies that 

Substituting this formula for <^j(C) in the jth column of A^^\ and pulling out the integrals over w, 

detA(^)(C) = — / e^^^'"-''Hei{A'{w))Mdw (77) 

where the entries of A'{w) are A'^^^{w) = T{a + v^j'p'^^ where p^, = Mtt^ when h ^ j and p;, = Mw 
when b = j. Hence 

fc^j ^ '^^ k=l "^^ l<a<b<Ar ^ ^ 

and so using (|H5|l . 

But for any simple closed contour T that encloses 7ri,...,7rAr but excludes w, and is oriented 
counter-clockwise, 



T r 1 ^ ^ 

2m Jr it; — -Zji-z — TT/c rr. tt,- — vr^ 



fc=l j=l k^j 

Therefore, we find (note N + u = M) 

N 



{CA-'B){rj,C) = fdz [ d«;e-''^^(-^)+f*^(-^)^ TT ^^^^ • (- f 

[2my Jy "w-z^^^z-TTk ^w^ 



(81) 



which completes the proof when all vr^'s are distinct. When some of vTj's are identical, the result 
follows by taking proper limits and using the I'Hospital's theorem. □ 

Note that for z G T and w £ Ti, Re{w — z) < 0. Hence using 

1 r°° 

= -M e^*^("'-'^-(^-''»(i?/ (82) 



w — z 
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for l/{w — z) in (jHH), the kernel 'K.M^Niv^ C) is equal to 

KM,Nin,C)= H(r? + y)J(C + y)d2/ (83) 
Jo 

where 

N 

H(7y + y) = — [ e-(^+2^)A-^(--«)z*'^ fr -^dz (84) 

and 

N 

J(C + y) = — /" e(^+3')*^("'-5)u;^^^ ff (vr^ - t/;)du;. (85) 
2^ k=i 

2.2 Asymptotic analysis: basic ideas 

From now on, as mentioned in the Introduction, we assume that 

TTr+l = ■ ■ ■ = TTN = 1. (86) 

In this case, dHU, (jHH) and (jHSJ) become 

POO 

KM,NivX)= il{r] + y)J{C + y)dy (87) 
Jo 



where 



and 



-— f e-*-^^(^-^) - TT -^—dz 

^^^"27ryr (1 - ^)^- n vr, - 

•^^ fc=l 



Set 



^ = 7' > 1. (90) 

For various choices of vTj, 1 < j < r, we will consider the limit of P(Ai < ^) when ^ is scaled as of 
the form (see Theorem II .11) 

? = ri+ (91) 

for some constants /x = /i(7), = i^(7) and for some a, while x is a fixed real number. By translation 
and scaling, the equation H63() becomes 

Ai < + —J = det(l - KM,iv|(^+^,oo)) = det(l - /Cm,7v) (92) 
where 1Cm,n is the operator acting on L^((0,C!o)) with kernel 

/CM,^(n, v) = —Y.M,N [ii + + ^-J^) ■ (93) 
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Using (|57j) . this kernel is equal to 



/•oo 

/ n{x + U + y)J{x + V + y)dy (94) 
Jo 



where 



and 



We need to find limits of ]Cm^n{u^v) for various choices of tTj's as M,N oo. A sufficient 
condition for the convergence of a Fredholm determinant is the convergence in trace norm of the 
operator. As JCm,n is a product of two operators, it is enough to prove the convergences of 7i and 
J' in Hilbert-Schmidt norm. Hence in sections El and 0] below, we will prove that for proper choices 
of fi, V and a, there are limiting operators Woo and Joq acting on -L^((0, cxd)) such that for any real 
a; in a compact set, 

/"OO /"OO ^ 

/ / \ZM'HM,N{x^u^y)-'Hooix^u^y)\ dudy (97) 
Jo Jo 

and 

/ / \^JM,N{x + u + y)-Jooix + u + y)\^dudy^O (98) 

Jo Jo 

for some non-zero constant Zm as M,N — > oo satisfying H90|) for 7 in a compact set. We will use 
steepest-descent analysis. 



3 Proofs of Theorem 11.11 (a) and Theorem 11.21 

We first consider the proof of Theorem ll.il fa). The proof of Theorem 11.21 will be very similar (see 
the subsection 13.41 below) . We assume that for some < k < r, 



1 + 7"^ (99) 



and TTf^^-^ > • • • > VTj. ^ are in a compact subset of (0, 1 + 7 ^). 
For the scaling (|91jl . we take 

a = 2/3 (100) 

and ^ 

= ^(^) := fl±2\ , ^ = ^(^) := il+j)^ (101) 
V 7 / 7 
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so that 



The reason for such choices will be made clear during the following asymptotic analysis. There is 
still an arbitrary parameter q. It will be chosen in H118|) below. 
The functions (|95|1 and (|96jl are now 

n{u) = / e-*^^/^«(-'?)e^^^(^) dz (103) 

^ -'r [p(. — z) g{z) 



and 



with 



j(^) = !^ f e-MV3,(,_,)^_M/(.). _ ,fg^,)dz (104) 



s 



and 



Pc := — ^, (105) 
7 + 1 

f{z) := -ii{z -q)+ \og{z) - \ log(l - z), (106) 

T 

where we take the principal branch of log (i.e. log(z) = ln|z| + iarg{z), —it < arg{z) < vr), and 

1 

^ e=k+i 
Now we find the critical point of /. As 



f'{z) = is a quadratic equation. But with the choice (|101|) of fi, there is a double root at 
z = Pc = Note that for 7 in a compact subset of [l,oo), pc is strictly less than 1. Being a 

double root, 

f'iPc) = f"{Pc) = 0, (109) 

where 

^ [z — ly 

It is also direct to compute 

/^^H^) = 4-^^^, /(3)(p,) = ^(l±l)! = 2.3 (111) 

^s, 7^(z — Ij'^ 7'^ 



and 

^7^) "7^°^ (7+1 



/(Pc) = -/i(Pc -q)+ log ( ) - 71 log ( -TT-^ ] ■ (112) 
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As f^^\pc) > 0, the steepest-descent curve of f{z) comes to the pomt pc with angle ±7r/3 to 
the real axis. Once the contour T is chosen to be the steepest-descent curve near pc and is extended 
properly, it is expected that the main contribution to the integral of 7i comes from a contour near 
Pc- There is, however, a difficulty since at the critical point z = pc, the integral (|1U3() blows up 
due to the term {pc — z')^ in the denominator when /c > 1. Nevertheless this can be overcome if we 
choose r to be close to pc^ but not exactly pass through zq. Also as the contour T should contain 
all vTj's, some of which may be equal to we will choose F to intersect the real axis to the left of 
Pc- By formally approximating the function / by a third-degree polynomial and the function g(^z') 
by ^(Pc), we expect 



n(u)^^^^^^, / e-^^^^ ' ''^^-''^e'H^^^^^^^^^^-P^^^ —^dz (113) 

27r g(pc) J r {pc-z) 

for some contour Too- Now taking the intersection point of T with the real axis to be on the left of 
Pc of distance of order M~^/^, and then changing of the variables by I'M^^'^^z — Pc) = a, we expect 

f-l/Ml/3)fcgM/(p,) r , 1 3 1 

niu) ^ ^ ^-uM^Mp^-i) e-^^^+h'^da (114) 

2vrs'(pc) Jroo « 

Similarly, we expect that 

■^<"> ~ Sf^'""""'""" L 

for some contour Eqo- When multiplying 7i{u) and J{v), the constant prefactors 

and 1/-^A/ cancel each other out. However, note that there are stih functions e-^*'^'^'^(P--'?) and 
^uM'^''^v{pc-q) ^ Qj^g q£ which may become large when M — > oo (cf. ()97|) . ()98() ). This trouble can be 
avoided if we can simply take q = Pc, but since T should be on the left of q, this simple choice is 
excluded. Nevertheless, we can still take q to be pc minus some positive constant of order M~^^^. 
Fix 

e > (117) 

and set 

We then take T to intersect the real axis at pc — c/ {vM^/"^) for some < c < e. With this choice 
of q and T, S, we expect that 

ZMHiu) ^Hooiu) (119) 
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where 



and 



where 



Ti-ooiu) 



27r 



Z 



M 



-J{v) ~ JUv) 



Jooiv) :-- 



,1„3 



a da. 



(120) 
(121) 
(122) 



Here the contour Too is, as in the left picture of Figure IHl from ooe^^^^ to ooe~*'^/^, passes the real 
axis on the left of the origin and lies in the region Re{a + e) > 0, is symmetric about the real axis 
and is oriented from top to bottom. The contour Sqo is, as in the right picture of Figure El from 
^g-«27r/3 ^gi27r/3^ jjgg ^^iq region Re{a + e) < 0, is symmetric about the real axis and is 
oriented from bottom to top. 




271/3 



Figure 3: Contours Fqo and Sqo of 7i{u) and J{v) 

This argument should be justified and the following is a rigorous estimate. 

Proposition 3.1. Fix e > and set q by (fTTH)) . Define 7ioo{u) and Joo{v) by (fHU)) and (HHJ), 

respectively. Then the fallowings hold for Ti.{u) and J{v) in (|lU3jl and (jlU4() for M/N = 7^ with 
'J in a compact subset of [1, 00). 

(i) For any fixed C/ G M, there are constants C,c > 0, Mq > such that 



IZMT-Ciu) - 7iociu)\ < 



MV3 



(123) 



foru>U when M > Mq. 
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(ii) For any fixed V gW, there are constants C, c > 0, Mq > such that 

\-J(„)-J^M\<^ (124) 

forv>V when M > Mq. 
The proof of this result is given in the following two subsections. 

3.1 Proof of Proposition |0 (i) 

The steepest-descent curve of / will depend on 7 and M. Instead of controlling uniformity of the 
curve in 7 and M, we will rather explicitly choose T which will be a steep-descent (though not the 
steepest-descent) curve of f{z). Fix R > such that 1 + -R > max{l, vTr+i, . . . , vTfc}. Define 

To := {Pc + ^^^e^' ■■7t/3<9<7t} (125) 

Ti := {p, + te^-/3:_-l^<t<2(l-p,)} (126) 

Fa := {pc + 2(1 - Pc)e'''/^ + x : < x < R} (127) 

Fg := {l + R + iy:0<y<V3il-p,)}. (128) 

Set 

F = (uLorO U (ul^oFfc) (129) 
and choose the orientation of F counter-clockwise. See Figure H) 



1 






, -c 

To / 







1 

q VPc 1 


- 

1+R 







Figure 4: Contour F 
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Note that all the singular points of the integrand of Ti are inside of T and hence the deformation 
to this new F is allowed. Direct calculations show the following properties of T. Recall 

f{z) := -fi{z -q)+ \og{z) - \ log(l - z). (130) 

T 

Lemma 3.1. For 7 > 1, Re{f{z)) is decreasing for z & T1UT2 as Re{z) increases. Also for 7 in 
a compact subset of [l,oo), we can take R> large enough such that 

maxi?e(/(z)) < i?e(/(p*)). (131) 

where p^, := Pc + 2(1 — Pc)e*'^/'^ is the intersection of Ti and T2- 
Proof. For Fi, by setting z = pc + ^e'''/^ < t < 2(1 - pc), 
Fi{t) ■.= Re{f{pc + te'^''')) 



-fi{Pc + \t - q) + \Hpl+p,t + t^) - ^\n{{l- p,f -{I- p,)t + t') 



(132) 



'''''^ , t^((7 + l)^t^- (7' -1)^ + 27) 

2^^{pl+p,t + t^){{l-p,Y-{l-p,)t + t^y ^^^^^ 

The denominator is equal to 27^|zp|l — zp, and hence is positive. To show that the numerator is 
positive, set 

Ti{t) := (7 + 1)2*2 _ (^2 _ ^ 27. (134) 

A simple calculus shows that 

mi„ T.(0=|'''<^'' (135) 
K[0.2(i-,.)1 l_Ti(2(l-p,)), T>5. 

But Ti ( 2(Ty^/^ij ) ^ 2 for 1 < 7 < 5, and ri(2(l — pc)) = 6 for 7 > 5, and hence we find that 
Ti(t) > for t E [0, 2(1 — pc)] and for all 7 > 1. Thus we find that Fi{t) is an increasing function 
inte [0,2(1 -p,)]. 

For F2, by setting z = pc + 2(1 — pc)e^'^^^ + x, x >0, 

F2{x) := Re{f{p, + 2(1 - p,)e'^l'' + x)) 

1 1 (136) 

= + x-q) + - ln((l + xf + 3(1 - pcf) - ^ ln(x2 + 3(1 - p^)') , 

and 

F'(t\= -^2(3;) 

^ 72(7 + l)2((l+x)2 + 3(1 -Pe)2)(x2 + 3(1 -p,)2) ^ ' 
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where 



T2{x) =(7 + l)'^x^ + (7^ + 67=^ + 1272 + IO7 + 3)x3 

+ (27^ + 137^ + 2O7 + 9)x2 + 2(272 + 77 + 5)x + 6(7 + 2) > 0, x> 0. 

Hence F2{x) is decreasing for x > 0. 

For Ts, setting z=l+R + iy,0<y< ^3(1 - Pc), 

F3{y) = Re{f{l + R + iy)) 



(138) 



-^{R+l-p,) + ^ ln((l + Rf + y') - ^ ln(ii' + y') 



(139) 



(140) 



□ 



As 1 < /i = (7 + 1)V7^ < 4 for 7 > 1 and 1 - = 1/(7 + 1) < 1, F^iy) can be made arbitrarily 
small when R is taken to be large. But 

Reifip*)) = Re{f{pc + 2(1 - Pc)e^'^/3)) 

= -^J^ - /^(l -Pc) + l HpI + 2Pc(l - Pc) + 4(1 - Pcf) - ^ ln(5(l - p,f) 
is bounded for 7 in a compact subset of [1, 00). Thus the result follows. 
As 7 is in a compact subset of [1, 00), we assume from here on that 

1 < 7 < 70 (141) 

for a fixed 70 > 1- Now we split the contour F = T' U F" where F' is the part of F in the disk 
N — Pel < 5 for some 5 > which will be chosen in the next paragraph, and F" is the rest of F. Let 
F^ be the image of the contour F' under the map z i>M^^^{z — Pc) and let F^ = Fqo \ F^. Set 



niu) = n'iu)+n"iu), Uooiu) = n'^{u)+n'^{u) 



(142) 



where 7i'{u) (resp. 7i'^{u)) is the part of the integral formula of 7i{u) (res. 7ioo{u)) integrated 
only over the contour F' (resp. F'^^) 
Fix 6 such that 



< (5 < min| 



6C0 ' 2(1 + 70) 



}, Co := 43 + 4(1 +70)^ 



For \s — Pc\ < 5, 



[i-sY 



1 

< - 



4V(Pc-<^)4 (l-Pc-S)^ 
< 1(4^ + 2^(1 + 70)^)= Co. 



(143) 



(144) 
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Hence by the Taylor's theorem, for \z — pc\ < 5, 



Re{f{z)-f{pc)-^^-^{z-Pcf) 



3! 



< max 



|s-Pc|<5 4! 



\Z - Pc 



< Co\z-pc\^ < —\z-pc 
D 

Therefore, by recalhng we find for < t < 

i?e(/(p, + te-/3))-/(p,)<-^t3^ 
Especially, from the Lemma ITT] fnote that Re{f{z)) = Re{f{'z))), 

maxi?e(/(z)) < Re{f{pc + de'^/^)) < f{pc) - '^6^. 



(145) 



(146) 



(147) 



In the following sub-subsections, we consider two cases separately; first when u is in a compact 
subset of M, and the other when u > 0. 



3.1.1 When u is in a compact subset of M. 

Suppose that |n| < uq for some uq > 0. First we estimate 

\9{Pc)\ 



\ZMn"{u)\ < 



27r(i/MV3)fc-i 



^-uM^^uReiz-q) ^MRe{f{z)-f{pc)) 



1 



Pc - zr\g{z) 



-\dz\. (148) 



Using (HiZl) and i?e(z - q) < ^/(TTKfTS for z G T" , 

where Lr is the length of F and > is a constant such that 

1- < ^m\g{z)\ < C,. 
For 7 G [l,7o]) Lr, Cg and |5'(pc)| are uniformly bounded, and hence 



\ZMn"iu)\ < 6-12 



5^M 



when M is sufficiently large. 

On the other hand, for a G T'', we have a = te^™^^ ^ 5vM^/'^ < i < +oo, and hence 



\n'L{n) 



< 





1 










gEMO TOO 




JsvM^rA 





da 



< 



2tt 



gno|a|g^Re(a3) 



-\da\ 



-dt < e~~6 



(149) 



(150) 



(151) 



(152) 
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when M is sufficiently large. 

Now we estimate \Zm'H'{u) — e~^"'H^(u)|. Using tlie change of variables a = v M^/^{z-pc) for 
the integral (IT^ for n'^{u), 



\ZMn'{u)-n'^{u) 



< 



-uM^/^uRe(z-q) 



27r Jr' \iyM^/^{z -pc)\'' 



,Af(/(^)-/(pc) 



\dz\. 



(153) 



We split the integral into two parts. Let V = T'^ U where T'^ = Fq U Fq and F2 = F' \ F'^. 
For z E F'l, \z-pc\ = e/(2i/Af ^Z^). Hence using (|145j) . 



< max(|e*-^(/(^)-/(^'=))|, |e^V(-Pc)3|) _ ^^^^^^ _ ^^^^^ _ 



(154) 



< ei6 



Also 



where 



giPc) 



< 



max ^'(s)! • \z — Pc\ < 



CpCe 
2zyMi/3 



(155) 



C := max{|5r'(s)| : \s - Pc\ < 



,SEF'2} 



2I/M1/3 

which is uniformly bounded as F is uniformly away from the singular points of g. Hence using 

^M(f(z)-f(p,) 9{Pc) _ M!^{z-p,f 
9{z) 



(156) 



,M{f{zyf(p,) _ ^M4(z-Pcf\ 9iPc) ^ ^M^{z-p,f l 9{Pc) 

9{z) 9{z) 



(157) 



< 



A 

V 16i/"4 



(C^ CoC\ _J_ 

+ 21/ ■ MV3' 



and the fact that the length of F'^ is 47ri?o/(3M^/^), we find that the part of the integral in ()153() 
over T'^ is less than or equal to some constant divided by M^/^. 

For z G F'2, we have z = pc + te^'""'^, eji^vM^I'^) <t<6. From (tnK|l (cf. (HSl) 



|gM(/(^)-/(p,)) _gMV(^-Pc)3| 



(158) 
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Also 



and hence 



g{pc) 



9{z) 



< ] max \g\s)\ ■ \pc - z\< CoCt, 
\giz)\ ser'^ 



(159) 



+ 



^ g{z) > 



Using 



9{z) 

9{z) 



uM^/^uRe{z-q) ^ ^vM^/^uo(\pa-q\ + \z-pq\) _ ^<iUo+vuoM^/''^t 



(160) 



(161) 



we find by substituting z = pc + te^™f^ into 1)153^ , the part of the integral in H153() over T'2 is less 
than or equal to 



vr 



(-5 euo+vuoM^/'-^t 3 

-{C^ + CoC) I ^77T7T75TV^e-VA«'(Mt4 + t)dt. 



(162) 



Then by the change of variables s = vM^/^t, the last integral is less than or equal to 



lc3 



(163) 



which is a constant divided by M^/^. Therefore, together with the estimate for the part over T'^, 
this implies that 



\ZM'H'{u)-n'^{u)\ < 



Ci 



(164) 



Ml/3 

for some positive constant Ci > for any M > 0. Now combining ()151() . ()152() and 1)164(1 . we find 
that for any uq > 0, there are constants C > 0, Mq > which may depend on uq such that 

C 



\ZMn{u)-noo{u)\ < 



MV3 



(165) 



for l-ul < no and M > Mq. obtain (fT^ . 



3.1.2 When u > 0. 



We first estimate \Zm'H"{u)\ using (|TIH|) . For z G F", i?e(z - g) = Re(j)c - q) + Re{z - Pc) > 
+ ^S, and hence as n > 0, 



\Zm'H"{u)\ < 



\9{Pc)\ 



27r(i/MV3)fc- 



(166) 
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On the other hand, for a E Fj^, by estimating as in ()152|) but now using n > and Re{a) > 
^6uM^/'-^ for a G r'4, we find 

|W'4(n)| < ^ / — \da\ < / -^^\da\ 

2tt 7r'^ l«r 27r 7rj4, (i67) 

when M is sufficiently large. 

In order to estimate \Zm'H'{u) — TC'^{u)\, we note that as n > 0, for z £ T' , 



-uM^/^uRe{z-q) 



< (168) 



which is achieved at 2; = — 2uM^/'i • Using the same estimates as in the case when u is in a 
compact set for the rest of the terms of (|153j) , we find that 

\ZMn'{u)-n'M\<^^ (169) 

for some constant C2 > when M is large enough. Now ()166() . (|167() and 1)169^ yield 1)123^ for the 
case when it > 0. 

3.2 Proof of Proposition lO (ii) 

We choose the contour S explicitly, which will be a steep-descent curve of —f{z), as follows. Let 
R>0. Define 

51 := {p, + : < t < 2p J (171) 

52 := {pc + 2pce^*^/^ -x:0<x<R} (172) 

53 := {-i? + i(^Pc-y) :0<y < ^Pe}, (173) 

and set 



S= (uLoSfc)u(ul^oSfc). (174) 

The orientation of S is counter-clockwise. See Figure El We first prove decay properties of 
Re{—f{z)) analogous to Lemma IXTl 

Lemma 3.2. For 7 > 1, Re{—f{z)) is decreasing for z G Si U S2 as Re{z) decreases. Also for 7 
in a compact subset of [l,cxo), we can take large R> (independent ofj) such that 

maxi?e(-/(z)) < Re{-f{p,))). (175) 
^£0-3 

where p^, = Pc + Ip^e^™!'^ is the intersection ofT,i and S2. 
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Figure 5: Contour S 



Proof. For z € Si, by setting z = Pc + te^™!'^ , < t < 2pc 



1 

2 



m(Pc - - c) - - ln(p^ - + * ) + ^ - pe)' + (1 - Vc)t + t 



and hence 



272 (p2 _ + t2) ((1 _ p^)2 + (1 _ + ^2) ' 

which is non- negative for ah t > 0. 

For 112, by setting z = + 2pce^*'^/^ - x, x > 0, 

F2(x) := Re{-!{j>c + 2p,e2-/3 - x)) 

= /i(-x-c)- iln(x2 + 3p2) +-Lln((l+x)2+3p2). 



A direct computation shows that 
where 



r2(:r) 



72(7 + 1)2 (x2 + 3p2) ((1 + x)2 + 3p2) 



72 (x) =(7 + l)^x^ + (37^ + 107=^ + 1272 + 67 + l)x=' 

+ (97^ + 207^ + 137^ + 27)x2 + 2(57'' + 77^ + 272)x + 6(27^ + 7^ 

Hence F2{x) is decreasing for x > 0. 



(176) 



(177) 



(178) 



(179) 



(180) 



29 



For S3, setting z = —R + i{^/3pc — y), < y < V^Pc, 
F3(y) = Re{-fi-R + i{V3p, - y)) 

= - C) - i ln(i?2 + (^^^ _ y)2) + _L + i?)2 + (^p^ _ y)2) . 

When R — > +00, -F3(y) can be made arbitrarily small. But 

i2e(-/(pe + 2p,e2-/3)) 

= -^c - ln{V3pc) + ^ ln(l + 3pl) 



(181) 



(182) 



is bounded for 7 in a compact subset of [1, 00). Thus the result follows. □ 

Let 6 be given in 1)143^ . Let S = S' U T," where S' is the part of S that lies in the disk 
1-2 — Pc\ < ^, and let S" = S \ S'. Let be the image of S' under the map z vM^/'^{z — Pc) 
and let S'^, = Sqo \ S'^^ . Set 

J{v) = J'(v) + J" (7;), Joo = J-<^(^;) + ^■^^^(t') (183) 

where J' {v) (resp. J!yo{v)) is the part of the integral formula of J{v) (resp. Jooiy)) integrated 
over the contour S' (resp. Sj^). 

As before, we consider two cases separately; first case when f is in a compact subset of M, and 
the second case when v > 0. 

3.2.1 When v is in a compact subset of R. 

There is i^o > such that \v\ < vq. First, we estimate 

\^j"{v)\ < (^^-^'^')'|' f e'^^^'''^'ol^-''le^'^«'^(-/(^)+/(^'=))|p,-zh5(z)||dz|. (184) 
Zm 2TT\g{pc)\ Jj]ii 

From Lemma 13.21 and ()145p , following the proof of 1)147^ , we find 

maxi?e(-/(z)) < Re{-f{p, + Se^'^'^) < f{p,) - '^5^ (185) 
Hence using I2: — g| < y'iRi + 1)2 + 3 for z e S", 

\^j"{v)\ < ^^^'f^'|' e-^^^^^-°V(^^^^^e-^^^^^^(5^C,Ls, (186) 
Zm 2TT\g{pc)\ 

where Cg is the maximum of \g{z)\ over z G T," and is the length of T," , both of which are 
uniformly bounded. Hence we find that when M is sufficiently large, 

' -J"iv)\<e-'^^'^. (187) 



Zm 
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When a G a = te=^2i7r/3^ 5vM^/^ < t < +00, and 



\J!L{v)\ 



2tt 



(188) 



when M is sufficiently large. 
Finally we estimate 



1 



< 



J'{v)-JUv)\ 



2-K 



-Mif{z)-f{pc)) 



9{z) 

g{pc) 



(189) 



\dz\. 



As before, we split the contour S' = S'^ USg where S'^ = SqUSq and S2 = and by following 

the steps of p54j) - p6:-{|) . we arrive at 



±-J'i.)-JUv)\<^,. 



M 



(190) 



for some constant C3 > when M is large enough. From l|187() . (|188|) and (|190p . we obtain 1)124^ . 



3.2.2 When v > Q. 

The proof in this case is again very similar to the estimate of T-i{u) when ti > 0. Then only change 
is the following estimates: 

e 1 



Re{z -q)< Re{pc + de^'""/^ - q) 



/MV3 2 



6, z £ S", 



(191) 



and 



Re{z -q) < Re{pc + 
Re{z — q) = 



3e 



1, 



„2i7r/3 



<1) 



2zyMV3' 



z e s', 



2'^ ^^' + i.Mi/3' 



Then for large enough M > 0, 



e 6 



and 



- MV3^ 



for some constant C > 0. We skip the detail. 



(192) 

(193) 

(194) 

(195) 
(196) 
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3.3 Proof of Theorem O (a) 

From the Proposition 13.11 the discussion on the subsection 12.21 imphes that under the assumption 
of Theorem ll.il (a). 

P((A, - (1 + j-^) . ^^^M^/^ < x) (197) 

converges, as M — > oo, to the Fredholm determinant of the operator acting on L^((0,oo)) whose 
kernel is 

/ noo{x + u + y)Joo{x + v + y)dy. (198) 
Jo 

From the integral representation (|lUjl of the Airy function, by simple changes of variables, 

Ai{u) = — [ e-""+^"'(ia = — [ e'^'^-^'^'db, (199) 
2tti Ir 27n l-p 

and hence a simple algebra shows that 

/•oo 

-Hooiu + y)Joo{v + y)dy - / e-^("+^Ui(n + y)Ai{v + y)e<"+yUy 
^ „—e(u—v) r r , , /'A 1 roo 



. „ 3 e 3 

A; 

= ^ e-™s('")(n)t('")(u)e™, 

m=l 

where the choice of the contours Sqo and Fqo ensures that i?e(a — 6) > which is used in the third 
equality. 

Let E be the multiplication operator by e~^"; (E/)(n) = e~^"/(ii). The computation (|2U(J() 
implies that H197() converges to 

k 

det(l - EA^.E"i - y e4™^ ii'^^E-i). (201) 

m=l 

The general formula of the Fredholm determinant of a finite-rank perturbation of an operator yields 
that this is equal to 

det(l - EA,E-i) • detf(5™„- < — —l—^Esi-), 4")E-i >] , (202) 

V 1-EA^E /l<ra,n<k 

which is equal to (|17() due to the proof of the following Lemma. This completes the proof. 
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Lemma 3.3. The function Fk{x) in Definition M.lX is well-defined. Also s^'^\u) defined in 
can he written as 

l'_1'\r!. 1 fu 



£+3n=m- 



Proof. It is known that A^,. has norm less than 1 and is trace class (see, e.g. jSHI)- The only 
thing we need to check is that the product < i„a s^"^\t^"'^ > is finite. By using the standard 
steepest-descent analysis, 

and 

tM(.)~^^e-P^ (205) 

But for s^™), since the critical point a = i is above the pole a = 0, the residue at a = contributes 
to the asymptotics and s^'^\u) grows in powers of ti as u — > +oo: 

s(-)(n)~ y t})lu' + —t^i^e-h'^\ u^+oo. (206) 
V ^ 3n£!n! 2^/^W2uV4 ^ ^ 

But the asymptotics ()204p of the Airy function as u — > cxd implies that for any U,V gM, there is a 
constant C > such that 

|A(u,t;)| < C7e"§("''" u>U, v>V, (207) 

which, together with (|2()5j) . implies that the inner product < i_a ■s^'"^ ^^"^ > is finite. 

Also s^^\u) defined in 1)13^ satisfies -^^s^'^\u) = Ai{u). Hence s^'^\u) is m- folds integral of 
Ai(u) from cxd to u plus a polynomial of degree m — 1. But the asymptotics ()206() determines the 
polynomial and we obtain the result. □ 

3.4 Proof of Theorem HH 

The analysis is almost identical to that of Proof of Theorem 11.11 (a) with the only change of the 
scaling 

^l' = ^ + ^^'-^r (208) 

We skip the detail. 
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4 Proofs of Theorem 11.11 (b) 

We assume that for some 1 < k < r, 

vrf 1 = • • • = vr-i > 1 + (209) 

are in a compact subset of (1 + 7"""^, 00), and vr^^-^, • • • , tt"^ are in a compact subset of (0, vrj"^). 
For the scaling ()91() . we take 

a = 1/2 (210) 

and 



so that 



1 7~^ / 1 7-2 

^ = ^(7):=— + ^ r, ^ = ^(7) := W — - 7^ (211) 

vri (1 - TTi) y vrf (1 - TTlY 

It is direct to check that the term inside the square-root of u is positive from the condition H209|l . 
Again, the reason for such a choice will be clear during the subsequent asymptotic analysis. 
The functions l|95p and (|96j) are now 

n{u) = / ,-.M^/^.(.-.),M/(.) 1 (213) 



27r 

and 

.7(v) = ^ 

27r 



r 



(vri - g{z) 



j^y) = '^MlH I ^uM^/Hz^,)^^Mfiz)^^^ _ ^fg{:,)dz (214) 



where 

f{z) := -fi{z -q)+ \og{z) - \ log(l - z), (215) 

T 

where log is the principal branch of logarithm, and 

3^^y-=T[^- n (^^-^)- (216) 

' e=k+i 

The arbitrary parameter q will be chosen in ()220|) below. Now as 

/(z) = -M + -- 2/ IV (217) 

z 7^(z — Ij 

with the choice 1)211^ of /U, two critical points of / are z = tti and z = -j^. From the condition 
H209p . it is direct to check that 

TTi < < — < 1. (218) 

1 + 7 /iVTi 
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Also a straightforward computation shows that 

/"(vri) = < 0, /"(— ) = (7^^m(l - vri))' > (219) 

Due to the nature of the critical points, the point z = tti is suitable for the steepest-descent 
analysis for J{v) and standard steepest-descent analysis will yield a good leading term of the 
asymptotic expansion of J{v). However, for 7i{u), the appropriate critical point is z = l/(/i7ri), 
and in order to find the steepest-descent curve passing the point z = l/dxiri), we need to deform 
the contour T through the pole z = tti and possibly some of TTk+i, . . . , VTr. In the below, we 

will show that the leading term of the asymptotic expansion of 7i{u) comes from the pole z = tti. 
Before we state precise estimates, we first need some definitions. 

Given any fixed e > 0, we set 

g := ^1 (220) 



Set 

Woo(^):=^e-^"-Res,=o(^e-^'^'-"'^), J-^(^) := i-e^ ^ s'^e^^'+^ds, (221) 

where is the imaginary axis oriented from the bottom to the top, and let 

_ (-l)fce-^/(-Off(vrO 

•= ^^J^2 • (222) 

Proposition 4.1. Fix e > and set q by (|220|) . The followings hold for M/N = 7^ with ^ in a 
compact subset of [l,oo). 

(i) For any fixed V there are constants C, c > 0, Mq > such that 

1 Ce~™ 
—J{v)-J^{v)\<^=- (223) 



for V >V when M > Mq . 
(ii) For any fixed J7 G M, there are constants C,c> 0, Mq > such that 



\ZMH{u)-noo{u)\< —=- (224) 



foru>U when M > Mq. 
We prove this result in the following two subsections. 
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4.1 Proof of Proposition 14.11 (i) 

Let R > and define 



^1 

S3 



{VTI 



2e 



+ : < y < 2} 



{vTi + 2i — X : 



2e 



<x<R} 



{TTi-R + i{2-y):0<y<2}, 



and set 



uUi^k) U (U 



The orientations of S,, j = 1, 2, 3 and S are indicated in Figure IHl 



q 71] 



(225) 

(226) 
(227) 

(228) 



Figure 6: Contour E 



inaxRe{-f{z))<Re{-f{p,)), 



Lemma 4.1. For 7 > 1, Re{—f{z)) is decreasing for z E Si U S2 as z travels on the contour along 
along the prescribed orientation. Also when 'j is in a compact subset of [l,oo), we can take R > 
large enough so that 

(229) 

where p^, = tti + 2i is the intersection of Si and S2. 

Proof. Any 2: G Si is of the form z = xq + iy, < y < 2, xq := tti — J^jj^ - Set for y > 0, 

Fi{y) := Re{-f{xo + iy)) = ^(xq H^l + /) + ^ ln((l - x^f + y^). (230) 



Then 



F[{y) 



-j/((7'-l)?/' + 7'(l-^o)'-xg) 

72(x2 + y2)((l_^0)2 + y2) 



(231) 
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But as < < VTi < a straightforward computation shows that 7^(1 — xq)^ — Xq > 0. 

Therefore, Re{—f{z)) decreases as z moves along Si. 



For z G S2, we have z = tti — x + 2i, ^t-^ < x < R. Set 

uy Al 



F2{x) :=i?e(-/(7ri-x + 2z)) =^f(^i-g-x)-Jln((7ri-2;)2 + y2) + ^ln((l-vri+x)2 + y2). (232) 
Then 

, X — TTi X + 1 — TTi 

-f^(^)=~/^~7 75 7 "I 577 ^ 75 77- (233) 

^ ^ (x-7ri)2 +4 72((x + l-7ri)2+4) ^ ^ 

As the function (7(5) = ^^^^ satisfies — j < ^(s) < 5 for all s € M, we find that for all x G M, 

^; / N 11 4 — 771 3 + 771 , , 

i1W<-M + j + j^ = -^-j^,(^ (234) 

using the definition l|211j) of /x. But as < tti < < 1, -F2(^) < x G M, and we find that 

Re{—f{z)) decreases as z moves on S2. 

For z G S3, z = TTi — R + i{2 — ?/), < y < 2. Then for 7 in a compact subset of [1, 00), we can 
take R > sufficiently large so that 

Fsiy) ■.= Re{-fi7ri-R + i{2-y)) 



1 1 (235) 

can be made arbitrarily small. However 



/i(7ri -R-q)-^ ln((7ri - Rf + (2 - yf) + ^ ln((l - 7ri + Rf + (2 - yf) 



Re{-fi7Ti + 2i)) = ^(771 -q)-\ ln(77? + 4) + ^ ln((l - 771)^ + 4) (236) 

is bounded for all 7 > 1. Hence the result ()229() follows. □ 

As 7 is in a compact subset of [1, 00), we assume that 

1 < 7 < 70 (237) 

for some fixed 70 > 1- Also as 7ri is in a compact subset of (0, 7^), we assume that there is 
< n < 1/2 such that 

n < 771 (238) 

Fix S such that 

H 1 1/2 1 „ 8 / 1 



Then for I2; — 77i| < 5, by using the general inequality 

|i?e(-/(.)+/(77i) + i/"(77i)(z- 771)1 < (|^niax^l|/(3)(,)|)|z-^i|3 (240) 
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and the simple estimate for |s — 7ri| < 5, 

2 



g3 7^(5 — 1)3 

2 2 



16 128 

< 

we find that 



\Rei-fiz) + /(^l) + ^r(7ri)(z - TTi)] < Cl|z - TTi 



(242) 

< — |z — VTll^, l-Z — VTll < (5. 

We spht the contour E = S' U E" where S' is the part of S in the disk |-z — vr| < 6, and S" is the 
rest of S. Let S'^^ be the image of T,' under the map z i/\/M(z — tti) and let = Sqo \ ^'oo- 
Set 

J(^;) = J'{v) + J"(t;), Joo(^^) = JLiv) + J-c^^C^^) (243) 

where J'{v) (resp. i74,(^^)) is the part of the integral formula of J^{v) (resp. Joo{v)) integrated 
over the contour T,' (resp. T,'^). 

Lemma 14.11 and the inequality 1)242^ imply that 

maxi2e(-/(z) + /(^i)) < Re{-f{zo) + /(vn)) 

< Re{-^f"{7ri){zo - vri)2) + ^|zo - vrip (244) 

= i?e(V(zo-^i)^) + ^5^. 

where zq is the intersection in the upper half plane of the circle |s — 7ri| =6 and the line Re{s) = 
TTi — As M — > 00, zq becomes close to vri + i5. Therefore when M is sufficiently large, 

maxi?e(-/(z) + /(^i)) < -^^^^ ^245) 

Using this estimate and the fact that Re{z — vri) < for z E S", an argument similar to that in 
subsection 13.21 yields 1)223^ . We skip the detail. 

4.2 Proof of Proposition 14.11 (ii) 

By using the Cauchy's residue theorem, for a contour V that encloses all the zeros of g but vri, we 
find 

2vr Jp, (vri - zYg{z) 
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Using the choice (|'22U() of q and setting z = tti + for the residue term, we find that 



Zn,n(u) =m(u) + ^^^^^^ /" e-'^^"(^-^i)e^^(^(^)^-^(^i)) - dz (2A7) 

ZMn[u) «iW + 2^(^^^,_iy^/ e (z-vri)^5(^)^"- ^^^^^ 

where 

n,{u) := ze— Res.-of^e-"'^e^^(^(-+^)-^(-))-^^^^y (248) 

We first show that 7ii{u) is close to TLooiu). Note that all the derivatives /'-^^(vri) and (/^^^(vri) 
are bounded and |5(7ri)| is strictly positive for 7 and vri under our assumptions. The function 

e^(/(-i+^)-/(-i)) (249) 

has the expansion of the form 

e-h'+-'{-AvM)MvMy+-) (250) 
for some constants Cj's when a is close to 0. On the other hand, the function 



5(7ri) 

has the Taylor expansion of the form 



for different constants Cj's. Hence we find the expansion 



^(-1 + ^) 

^00 00 



for some constants Ci,di. Now as 



is a polynomial of degree at most i — 1 in u, we find that 



k—1 

\ / ,=1 



(251) 



l + ci(-^)+C2(^)' + --- (252) 



(253) 



Res,=o( ^e-*^"-^"' ) (254) 



(255) 
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for some polynomials qj. Therefore, due to the factor e in Tli, for any fixed [/ G M, there are 
constants C, c, Mq > such that 



< 



Ce- 



M 



(256) 



for all u>U when M > Mq. 

Now we estimate the integral over T' in (|247j) . We will choose T' properly so that the integral is 
exponentially small when M — > oo. Let tt^, := min{7r/j,_|_i . . . , vr^, 1, -j^}- Then tti = • • • = Tr^ < vr*. 
Let 5 > and R > max{7rfc_|_i, . . . , TTr, 1} be determined in Lemma 14.21 below. Define 



Ti 

r2 



r , ^1 + ^* ^ ^ i 
jx + : < X < xoj 



{1 + 



1 



o<f<^} 



1 + 7 

{x + i—^ ■.l<x<R} 

1 + 7 

{i? + ^(3^-y):0<y<^} 



where xq and 6q are defined by the relation 



xo + i(5 = 1 + 



1 + 7 



Set 



(257) 
(258) 
(259) 
(260) 
(261) 

(262) 
(263) 



r' = (u5^ir,)u(u5^ir,). 

See Figure [7| for F' and its orientation. 

Lemma 4.2. For 7 m a compact subset of [1, 00) and for vri in a compact subset of (0, jp^), there 
exist 5 > and c > such that 



i?e(/(z) - /(vri)) < -c, zGFiUF 



2- 



Also Re{f{z)) is a decreasing function in z G F3 U F4, and when R > max{7rfc_|_i, . , 
sufficiently large, 



Re{f{z) - /(vTi)) < Re{ /(I + i-^) - /(^i) ) , z G F5. 



Proof. Note that 



1/(^)1 



^1 7"^ 



2 z — 1 



(264) 

,7rr,l} is 



(265) 



(266) 
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Figure 7: Contour S' 



is bounded for z in the complex plane minus union of two compact disks centered at and 1. For 
7 and vri under the assumption, tti and are uniformly away from and 1 (and also from j^)- 
Therefore, in particular, there is a constant Ci > such that for z = x + iy such that x G [vri, 
and y > 0, 

|i?e(/(x + iy) - /(x))| < + iy) - < max + • |y| < CM- (267) 

0<s<l 

On the other hand, a straightforward calculation shows that when z = x\s, real, the function 

i?e(/(x) - /(vri)) = -^(x - g) + In |x| - 7"' hi |1 - x| - /(vn) (268) 

decreases as x increasing when x G (tti, y^)- (Recall that vri and ^j^- are the two roots of -F'(z) = 
and < vTi < < < 1.) Therefore we find that for 2; = x + such that x E [(tti + 
7rfc+i)/2, 

i?e(/(z) - /(tti)) < Re{f{x) - f{m)) + Ci\y\ 

<i?e(/(^^l±^)-/(vri))+CiM. 

As TT* is in a compact subset of (vri, 1), we find that there is ci > such that 

Re{f{z)-f{Tri))<-ci + Ci\y\ (270) 



for above z, and hence there are 5 > and c > such that for z = x + iy satisfying \y\ < 5, 

Reifiz) - /(vTi)) < -c. (271) 
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Note that as is in a compact subset of under out assumption, we can take 6 small 

enough such that xq defined by l|262|) is uniformly left to the point -j^. Therefore (|27H) holds for 
z G Si U S2. 

For z = 1 + Yp^e^^'^-^) G T3, 

FsiO) :=Re{f{l + -^e^(-~'^)) 
1 + 7 

= -/^(l + YT^ -0)-q) + l ln(l + cos(7r -9) + ^^^) (272) 



7^ ^ 1 + 7 
We set t = cos(7r — 9) and define 

Git) := F3(0) = + - g) + i ln(l + + ^^^) + ^ ln(l + 7). (273) 

Then 

G'{t) = -/i— — + . ^ , (274) 

^' ^1 + 7 l + 2(l+7)-it+(l+7)-2 ^ ^ 

is a decreasing function in t G [— 1, 1] and hence 

G'(t)<G'(-l) = ^(-A.+ (i^)',) < (275) 

as the function /x = — + — ^ in vri G (0, tt— 1 takes the minimum value ^"^^7^ at vr = -rr— • 
Therefore, G(t) is a decreasing function in t G [—1,1] and Re{f{z)) is a decreasing function in 

Set yi = For z G z = x + iyi, x > 1. Let 

F^ix) := Re{fix + %)) = -/x(x - ?) + ^ ln(x2 + yj) - 2_ ((x - 1)^ + yf) . (276) 

Then 

But the last term is non- negative and the middle term is less than 1 as a; > 1. Also by the 
computation of ()275() . fi > ^"^^7'' > 1. Therefore we find that -^4(2;) < for a; > 0, and F^ix) 
decreases as x > 1 increases. 

Finally, for z = R + iy, < y < j^, 

ReifiR + iy)) = -fi{R -q) + l H^^ + " ^ " + (^78) 
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can be made arbitrarily small when R is taken large enough, while 

Re{f{l + ^^^)) = -Ml - + ^ ln(l + - In ) (279) 

is bounded. □ 



This lemma implies that 

i?e(/(z) - /K)) < -c (280) 

for all z E F'. Also note that i?e(z — tti) > for z G F'. Therefore for any fixed [/ S M, there are 
constants C, c, Mq > such that 



^VMu(z-7Ti)^M(,f(z)-f(7ri)) ^ 



< Ce-^"e-^*^ (281) 



27r(i/^/M)fc-i ir' (-^ - 7ri)'=5f(2;) 

for M > Mq and for u>U. Together with 1)256^ . this implies Proposition 14.11 (ii). 

4.3 Proof of Theorem O (b) 

From the Proposition l4. H and the discussion in the subsection l2.2| we find that under the assumption 
of Theorem ll.il (h) 

(Ai - (-^ + -^)) • , ^ , < x) (282) 



^ (l-TTl)^ 

converges, as M — > oo, to the Predholm determinant of the operator acting on i>^((0, oo)) given by 
the kernel 

/ rLoo{x + u + y)Joo{x + v + y)dy. (283) 
Jo 

Now we will express the terms TCooiu) and Joo{v) in terms of the Hermite polynomials. 
The generating function formula (see (1.13.10) of |2^) of Hermite polynomials H^, 



Y^ElMe = e^^'-'\ (284) 



n=0 

implies that 

-ie^m^u) = ReSa=o(y^e-h'-^-+y)-^ = izll!^^^p,_^(^), (285) 

where the orthonormal polynomial pk-i{x) is defined in (|31|) . The forward shift operator formula 
(see (1.13.6) of ^) 

H'^ix) = 2nHn-i{x) (286) 

implies that 

p'^{y) = ^Pfc_i(y), (287) 
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and hence 

-ie^"Woo(n) = ReSa=of^e-i'^'-("+J')'^') = i^lO^^p', (n) . (288) 
On the other hand, the Rodrigues-type formula (see (1.13.9) of |2l] ) 

Hn{x)=e-'(^-^y[e-^'] (289) 

imphes that 

(27r)V4V^ V^^U 
Sine the integral which appears in Joo{v) is equal to 

j s''e^'"+'"ds = l^j^ ' j e^^'+^^ds = i (291) 

we find 

e-^Voo(^) = {-lfi{2T:r^'^Vk\e-^'/^pk{v). (292) 

After a trivial translation, the Fredholm determinant of the operator ()283() is equal to the 
Fredholm determinant of the operator acting on L^((a;,oo)) with the kernel 

K2iu,v):= 'Hoo{u + y)Joo{v + y)dy. (293) 
Jo 

By (OHHl) and (IM . 

(u - ^;)ii:2(n, t;)e^("-^) = / (n + y) • + y)Pk{v + y)e-^''+y^"/^dy 

p'kiu + y)pk{v + y)-{v + y)e-^"+y^'/^dy. 
Note that p/. satisfies the differential equation 

p'kiy) - ypkiy) + kPkiy) = 0, (295) 

which follows from the differential equation 

H'^{x) - 2xH'Jx) + 2nHn{x) = (296) 

for the Hermite polynomial (see (1.13.5) of HH). Now use ()295() for the first integral of (|294() by 
and integrate by parts of the second integral by noting that {v + y)e~^^^y^^ = to 
obtain 

kpk{u + y)pk{v + y)e-(^+3')'/2^y 

.oo (297) 
-p'^{u)pk{v)e--"'^- / p'kiu + y)p'k{v + y)e-^^^y^"'^dy. 



(294) 
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Note that the terms involving p'l.{u + y) are cancelled out. Then integrating by parts the second 
integral and noting that p'f^{u + y) = -§^Pk{u + y), we obtain 

(n - v)K2{u,v)e<--^'> = -p',{u)pk{v)e-^' + Pk{u)p',{v)e-^' . (298) 
By using (|287|1 . this implies that 

K2{n,v) = ^e-^uPk{u)p,-,{v)-p,Mn)Pk{v)^-.y2^.._ ^299) 

U — V 

Therefore, upon conjugations, the Fredholm determinant of K2 is equal to the Fredholm determi- 
nant det(l — Ha,) in This completes the proof of Theorem ll.il (b). 



5 Samples of finitely many variables 

In this section, we prove ProDOsition ll.il 

We take M — > 00 and N = k. We suppose that 

TTl = ■ ■ • = TTfc. (300) 

Then from (|60|) . the density of the eigenvalues is 

k 

p(A) = iy (A)2 l[ e~"^'^^ Af ^^ (301) 

and hence 

<t) = ^ j^- V{yf n e~^'^-y^yf"^dyj (302) 



where 



rfc-l. 



C = ^ \,.. -. (303) 

By using the change of the variables yj = ^(l + 

pf Ai < - + = J'^Zi. r ■■■ r ViO' fr e-^^^{l + (304) 



As M ^ 00 while k is fixed, vr^^^M'^'/^gfcAf,^ ^ (2vr)''/2 J]*^-i(l + By using the dominated 
convergence theorem. 

Urn F((Xl-—)^Tl^/Mx] =Gkix). (305) 

M^oo \ TTi J 

The result ()51l) follows from (|305p and the fact that G;^. is a distribution function. 
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6 Last passage percolation, queueing theory and heuristic argu- 
ments 



There is a curious connection between complex Gaussian sample covariance matrices and a last 
passage percolation model. 

Consider the lattice points £ 'S? . Suppose that to each (i, j), i = 1, . . . , A^, j = 1, . . . , M, 
an independent random variable j) is associated. Let (1, 1) {N,M) be the set of 'up/right 
paths' TT = {{ik,jk)}k=i^~^ where (i^+i, j^+i) - {ikJk) is either (1,0) or (0, 1), and {hji) = (1, 1) 
and [iN+M-i, Jn+m-i) = {N,M). There are (^^^^]~^) such paths. Define 

LiN,M):= max ^ Xii,j). (306) 

If X{i,j) is interpreted as time spent to pass through the site L{N,M) is the last passage 

time to travel from (1, 1) to {N,M) along an admissible up/right path. 

Let TTi, . . . ,TT]\f be positive numbers. When X{i,j) is the exponential random variable of mean 
—j-f (the density function of X{i,j) is vTj 

j^^-w,Mx^ X > 0), it is known that L{N,M) has the 
same distribution as the largest eigenvalue of the complex Gaussian sample covariance matrix of 
M sample vectors of N variables (see ()61|)): for M > N, 

P(L(iV,M)<x) = ly^ ^ ^(,) -'"-"" nOngf-^^g.- (307) 

We emphasize that X{i,j), j = 1,2, ...,M are identically distributed for each fixed i. As a 
consequence, we have the following. Recall that 

TrJ^=ij- (308) 

Proposition 6.1. Let L{M,N) be the last passage time in the above percolation model with expo- 
nential random variables at each site. Let Ai be the largest eigenvalue of M (complex) samples of 
N X 1 vectors as in Introduction. Then for any a; € M, 

P(L(M, N) <x)= P(Ai(M, N) <x). (309) 

Formula ()307|) for the case of vri = • • • = vrjv was obtained in Proposition 1.4 of ^7V. The general 
case follows from a suitable generalization. Indeed, let Xi,yj € [0,1) satisfy < Xiyj < 1 for all 
i,j. When the attached random variable, denoted by Y{i,j), is the geometric random variable of 
parameter Xiyj (i.e. F{X{i,j) = k) = {I — Xiyj){xiyj)^ , k = 0,1,2,...), the last passage time, 
G{N,M), from (1, 1) to {N,M) defined as in (fSU^ is known to satisfy 

F{G{N,M)<n)= Y[{l-Xiyj)- sx{x)sx{y) (310) 

i,j>l X:\i<n 
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where the sum is over all partitions A = (Ai,A2,...) such that the first part Ai < n, and s\ 
denotes the Schur function, x = {xi,X2, ■ ■ ■) and y = (yi, 7/2, • • • )• This identity was obtained by 
using the Robinson-Schensted-Knuth correspondence between generalized permutations (matrices 
of non-negative integer entries) and pairs of semistandard Young tableaux (see, e.g. [SO], (7.30) 
of [0]). The normalization constant follows from the well-known Cauchy identity (see, e.g. [301) 

Y,sx{x)sx{y)= Hil-XiVj) (311) 

A i,j>l 

where the sum is over all partitions. Now set Xj = 1 — i = I, . . . , N , Xi = 0,i > N and yj = 1, 
j = 1, . . . , M, yj = 0,j > M, and n = xL. By taking L ^ 00, it is easy to compute that jY{i,j) 
converges to the exponential random variable X{i,j), while one can check that the summation on 
the right-hand-side of ()31U() converges to the right-hand-side of H3U7|) . and hence the identity ()307() 
follows. There are determinantal formulas for the right-hand-side of (|31U() (see e.g. [2B) E01)j 
some of which, by taking the above limit, would yield an alternative derivation of Proposition 12. II 

Another equivalent model is a queuing model. Suppose that there are tellers and M cus- 
tomers. Suppose that initially all M customers are on the first teller in a queue. The first customer 
will be served from the first teller and then go to the second teller. Then the second customer will 
come forward to the first teller. If the second customer finishes his/her business before the first 
customer finishes his/her business from the second teller, the second customer will line up a queue 
in the second teller, and so on. At any instance, only one customer can be served at a teller and 
all customers should be served from all tellers in the order. The question is the total exit time 
E{M, N) for M customers to exit from A^ queues. We assume that the service time at teller i is 
given by the exponential random variable of mean ^^r^y- Assuming the independence, consideration 
of the last customer in the last queue will yield the recurrence relation 

E{M, N) = max{E{M - 1, N),E{M, A^ - 1)} + e(A^) (312) 

where e(A^) denotes the service time at the teller A^. But note that the last passage time in the 
percolation model also satisfies the same recurrence relation 

L(M, A^) = max{L(M - 1, N),L{M, A^ - 1)} + X{M, N) (313) 

where A(M, A^) is the same exponential random variable as e{N). Therefore we find that E{M, N) 
and L{M, N) have the same distribution. Thus all the results in Introduction also applied to 
E{M, N). 

Now we indicate how the critical value £j = vrj"*^ = 1 + 7"-*^ of Theorem 1 1 . 1 1 can be predicted in 
the last passage percolation model. 



47 



First, when X{i,j) are all identical exponential random variables of mean 1, Theorem 1.6 of 
j21j shows that, as M,N oo such that M/N is in a compact subset of (0, oo), L{N,M) is 
approximately 

L{N, M) ~ L(M, iV) ~ ( + ViV)2 + (^^^±^^^0 (314) 

where xo denotes the random variable of the GUE Tracy- Widom distribution. On the other hand, 
note that when = 1, X(l,j) are independent, identically distributed exponential random vari- 
ables of mean , and hence the classical central limit theorem implies that the last passage time 
from (1, 1) to (l,xM) is approximately 



where g denotes the standard normal random variable. Note the different fluctuations which is due 
to different dimensions of two models. 

Now consider the case when r = 1 in Theorem 11.11 i.e. tt2 = tts = ••• = vrjv = 1; X{i,j), 
1 < J < is exponential of mean and X{i,j), 2 < i < N, 1 < j < M is exponential of mean 
jj. We take M/N = 7^ > 1. An up/right path consists of two pieces; a piece on the first column 
(1, j) and the other piece in the 'bulk', (i,i), i > 2. Of course the first part might be empty. We 
will estimate how long the last passage path stays in the first column. Consider the last passage 
path conditioned that it lies on the first column at the sites (1, 1), (1, 2), . . . (1, xM) and then enters 
to the bulk (i, j), i > 2. See FigureEl Then from ()315() and ()314|) . we expect that the (conditioned) 
last passage time is, to the leading order, 

f{x) =Tr-^x+{VT^ + -f~'^f. (316) 

It is reasonable to expect that the last passage time is the maximum of /(x) over x G [0, 1], to the 
leading order. An elementary Calculus shows that 

, ^/(0) = (l+7-^)^ ifvr-i< 1 + 7-1 

max jix) = < (317) 

l/(i-(^) = ;^ + i^> ifvr-> 1 + 7-1. 

When max/ occurs at x = 0, the last passage path enters directly into the bulk and hence the 
fluctuation of the last passage time is of of Af~^/'^ due to 1)314(1 . But if the max/(a;) occurs for 
some X > 0, then the fluctuation is M"^/^ due to ()315|) . which is larger than the fluctuation M"^/"^ 
from the bulk. Note that the value of max / in (|317() agrees with the leading term in the scaling of 
Theorem ll.il This provides an informally explanation of the critical value 1 + 7"^ of 7rj~i. 

If one can make this kind of argument for the sample covariance matrix, one might be able to 
generalize it to real sample covariance matrix. 
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Figure 8: Last passage percolation when r = 1 
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